Electronic device and method for displaying object associated with external electronic device on basis of position and movement of external electronic device

ABSTRACT

An electronic device according to various embodiments of the present disclosure includes one or more cameras having a designated field of view, a display, a communication circuitry and a processor wherein the procesor is configured to identify an external electronic device among one or more external objects included in the designated field of view using the camera, display a graphic object corresponding to the external electronic device on the display based on first location information of the external electronic device identified based at least on the image information obtained using the camera, and when the external electronic device is out of the designated field of view, display the graphic object on the display based on second location information of the external electronic device identified using the camera before the external electronic device is out of the designated field of view and information related to the movement of the external electronic device received from the external electronic via the communication circuitry after the external electronic device is out of the designated field of view.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a 371 of International Application No.PCT/KR2019/003336 filed on Mar. 22, 2019, which claims priority toKorean Patent Application No. 10-2018-0061384 filed on May 29, 2018, thedisclosures of which are herein incorporated by reference in theirentirety.

BACKGROUND 1. Field

Various embodiments relate to an electronic device and a method fordisplaying an object related to an external electronic device usinginformation about the external electronic device.

2. Description of Related Art

Due to the development of technology, various electronic devices forproviding services related to virtual reality (VR) have been provided.Various electronic devices include an electronic device for displayingmultimedia content related to virtual reality and an external electronicdevice for user input in the multimedia content related to virtualreality.

A user receiving a service related to virtual reality via an electronicdevice has a limited field of view (or scope of view). Due to thelimitation of the field of view, a separate external electronic devicefor displaying a user's input in the service related to the VR isprovided. In the electronic device, a method for naturally displayingthe external electronic device is required.

The technical problems to be achieved in this document are not limitedto the technical problems mentioned above, and other technical problemsthat are not mentioned are clearly understood by those skilled in theart from the following description.

SUMMARY

An electronic device according to various embodiments of the presentdisclosure includes one or more cameras having a designated field ofview, a display, a communication circuitry, and a processor wherein theprocessor, using the camera, identifies an external electronic deviceamong one or more external objects included in the designated field ofview, displays a graphic (visual) object corresponding to the externalelectronic device on the display based on first location information ofthe external electronic device identified based at least on imageinformation obtained through the camera, and displays the graphic objecton the display, when the external electronic device deviates from thedesignated field of view, based on second location information of theexternal electronic device identified through the camera before theexternal electronic device deviates from the designated field of viewand information related to the movement of the external electronicdevice received from the external electronic device through thecommunication circuitry after the external electronic device deviatesfrom the designated field of view.

A method according to various embodiments of the present disclosureincludes an operation of identifying an external electronic device amongone or more external objects included in a designated field of viewusing one or more cameras having the designated field of view,displaying a graphic object corresponding to the external electronicdevice through a display based on first location information of theexternal electronic device identified based at least on imageinformation obtained through the camera, and displaying the graphicobject on the display, when the external electronic device deviates fromthe designated field of view, based on second location information ofthe external electronic device identified through the camera before theexternal electronic device deviates from the designated field of viewand information related to the movement of the external electronicdevice received from the external electronic device through thecommunication circuitry after the external electronic device deviatesfrom the designated field of view.

An electronic device and a method according to various embodiments ofthe present disclosure provide an enhanced user experience (UX) bydisplaying a graphic object representing an external electronic devicealong with multimedia content for providing virtual reality (VR) usingat least one of at least one sensor included in the electronic deviceand at least one sensor of an external electronic device interlockedwith the electronic device.

The effects obtainable in the present disclosure are not limited to theabove-mentioned effects, and other effects not mentioned are clearlyunderstood by those skilled in the art from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an electronic device in a networkenvironment displaying an object related to an external electronicdevice based on a location and a movement of the external electronicdevice according to various embodiments.

FIG. 2 illustrates an example of a functional configuration of anelectronic device and an external electronic device according to variousembodiments.

FIG. 3 illustrates an example of a functional configuration of anelectronic device, a first external electronic device, and a secondexternal electronic device according to various embodiments.

FIG. 4A illustrates an example of a situation in which an externalelectronic device deviates from a field of view of a camera of anelectronic device according to various embodiments.

FIG. 4B illustrates an example for comparing a field of view of a cameraand a field of view of a user according to various embodiments.

FIG. 5A illustrates an example of operation of an electronic deviceaccording to various embodiments.

FIG. 5B illustrates an example of an operation of an electronic deviceaccording to various embodiments.

FIG. 6 illustrates an example of an external electronic device and agraphic object displayed corresponding to the external electronic deviceaccording to various embodiments.

FIG. 7 illustrates a time flow of a display displayed according to anoperation of an electronic device according to various embodiments.

FIG. 8 illustrates an example for comparing the effect of correctionusing sensor data according to various embodiments.

FIG. 9A illustrates an example of an operation of an electronic deviceaccording to various embodiments.

FIG. 9B illustrates an example of an operation of an electronic deviceaccording to various embodiments.

FIG. 10 illustrates an example of a graphic object for guiding alocation of an external electronic device displayed on a display of anelectronic device according to various embodiments.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an electronic device 101 in anetwork environment 100 according to various embodiments.

Referring to FIG. 1, the electronic device 101 in the networkenvironment 100 may communicate with an electronic device 102 via afirst network 198 (e.g., a short-range wireless communication network),or an electronic device 104 or a server 108 via a second network 199(e.g., a long-range wireless communication network). According to anembodiment, the electronic device 101 may communicate with theelectronic device 104 via the server 108. According to an embodiment,the electronic device 101 may include a processor 120, memory 130, aninput device 150, a sound output device 155, a display device 160, anaudio module 170, a sensor module 176, an interface 177, a haptic module179, a camera module 180, a power management module 188, a battery 189,a communication module 190, a subscriber identification module (SIM)196, or an antenna module 197. In some embodiments, at least one (e.g.,the display device 160 or the camera module 180) of the components maybe omitted from the electronic device 101, or one or more othercomponents may be added in the electronic device 101. In someembodiments, some of the components may be implemented as singleintegrated circuitry. For example, the sensor module 176 (e.g., afingerprint sensor, an iris sensor, or an illuminance sensor) may beimplemented as embedded in the display device 160 (e.g., a display).

The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 101 coupled with theprocessor 120, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 120 may load a command or data received fromanother component (e.g., the sensor module 176 or the communicationmodule 190) in volatile memory 132, process the command or the datastored in the volatile memory 132, and store resulting data innon-volatile memory 134. According to an embodiment, the processor 120may include a main processor 121 (e.g., a central processing unit (CPU)or an application processor (AP)), and an auxiliary processor 123 (e.g.,a graphics processing unit (GPU), an image signal processor (ISP), asensor hub processor, or a communication processor (CP)) that isoperable independently from, or in conjunction with, the main processor121. Additionally or alternatively, the auxiliary processor 123 may beadapted to consume less power than the main processor 121, or to bespecific to a specified function. The auxiliary processor 123 may beimplemented as separate from, or as part of the main processor 121.

The auxiliary processor 123 may control at least some of functions orstates related to at least one component (e.g., the display device 160,the sensor module 176, or the communication module 190) among thecomponents of the electronic device 101, instead of the main processor121 while the main processor 121 is in an inactive (e.g., sleep) state,or together with the main processor 121 while the main processor 121 isin an active state (e.g., executing an application). According to anembodiment, the auxiliary processor 123 (e.g., an image signal processoror a communication processor) may be implemented as part of anothercomponent (e.g., the camera module 180 or the communication module 190)functionally related to the auxiliary processor 123.

The memory 130 may store various data used by at least one component(e.g., the processor 120 or the sensor module 176) of the electronicdevice 101. The various data may include, for example, software (e.g.,the program 140) and input data or output data for a command relatedthereto. The memory 130 may include the volatile memory 132 or thenon-volatile memory 134.

The program 140 may be stored in the memory 130 as software, and mayinclude, for example, an operating system (OS) 142, middleware 144, oran application 146.

The input device 150 may receive a command or data to be used by anothercomponent (e.g., the processor 120) of the electronic device 101, fromthe outside (e.g., a user) of the electronic device 101. The inputdevice 150 may include, for example, a microphone, a mouse, a keyboard,or a digital pen (e.g., a stylus pen).

The sound output device 155 may output sound signals to the outside ofthe electronic device 101. The sound output device 155 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record, and the receivermay be used for an incoming call. According to an embodiment, thereceiver may be implemented as separate from, or as part of the speaker.

The display device 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display device 160 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaydevice 160 may include touch circuitry adapted to detect a touch, orsensor circuitry (e.g., a pressure sensor) adapted to measure theintensity of force incurred by the touch.

The audio module 170 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 170 may obtainthe sound via the input device 150, or output the sound via the soundoutput device 155 or a headphone of an external electronic device (e.g.,an electronic device 102) directly (e.g., wiredly) or wirelessly coupledwith the electronic device 101.

The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state(e.g., a state of a user) external to the electronic device 101, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 176 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 177 may support one or more specified protocols to be usedfor the electronic device 101 to be coupled with the external electronicdevice (e.g., the electronic device 102) directly (e.g., wiredly) orwirelessly. According to an embodiment, the interface 177 may include,for example, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connecting terminal 178 may include a connector via which theelectronic device 101 may be physically connected with the externalelectronic device (e.g., the electronic device 102). According to anembodiment, the connecting terminal 178 may include, for example, a HDMIconnector, a USB connector, a SD card connector, or an audio connector(e.g., a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus whichmay be recognized by a user via his tactile sensation or kinestheticsensation. According to an embodiment, the haptic module 179 mayinclude, for example, a motor, a piezoelectric element, or an electricstimulator.

The camera module 180 may capture a still image or moving images.According to an embodiment, the camera module 180 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to theelectronic device 101. According to one embodiment, the power managementmodule 188 may be implemented as at least part of, for example, a powermanagement integrated circuit (PMIC).

The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 101 and the external electronic device (e.g., theelectronic device 102, the electronic device 104, or the server 108) andperforming communication via the established communication channel. Thecommunication module 190 may include one or more communicationprocessors that are operable independently from the processor 120 (e.g.,the application processor (AP)) and supports a direct (e.g., wired)communication or a wireless communication. According to an embodiment,the communication module 190 may include a wireless communication module192 (e.g., a cellular communication module, a short-range wirelesscommunication module, or a global navigation satellite system (GNSS)communication module) or a wired communication module 194 (e.g., a localarea network (LAN) communication module or a power line communication(PLC) module). A corresponding one of these communication modules maycommunicate with the external electronic device via the first network198 (e.g., a short-range communication network, such as Bluetooth™,wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA))or the second network 199 (e.g., a long-range communication network,such as a cellular network, the Internet, or a computer network (e.g.,LAN or wide area network (WAN)). These various types of communicationmodules may be implemented as a single component (e.g., a single chip),or may be implemented as multi components (e.g., multi chips) separatefrom each other. The wireless communication module 192 may identify andauthenticate the electronic device 101 in a communication network, suchas the first network 198 or the second network 199, using subscriberinformation (e.g., international mobile subscriber identity (IMSI))stored in the subscriber identification module 196.

The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 101. According to an embodiment, the antenna module197 may include an antenna including a radiating element composed of aconductive material or a conductive pattern formed in or on a substrate(e.g., PCB). According to an embodiment, the antenna module 197 mayinclude a plurality of antennas. In such a case, at least one antennaappropriate for a communication scheme used in the communicationnetwork, such as the first network 198 or the second network 199, may beselected, for example, by the communication module 190 (e.g., thewireless communication module 192) from the plurality of antennas. Thesignal or the power may then be transmitted or received between thecommunication module 190 and the external electronic device via theselected at least one antenna. According to an embodiment, anothercomponent (e.g., a radio frequency integrated circuit (RFIC)) other thanthe radiating element may be additionally formed as part of the antennamodule 197.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronicdevice 104 via the server 108 coupled with the second network 199. Eachof the electronic devices 102 and 104 may be a device of a same type as,or a different type, from the electronic device 101. According to anembodiment, all or some of operations to be executed at the electronicdevice 101 may be executed at one or more of the external electronicdevices 102, 104, or 108. For example, if the electronic device 101should perform a function or a service automatically, or in response toa request from a user or another device, the electronic device 101,instead of, or in addition to, executing the function or the service,may request the one or more external electronic devices to perform atleast part of the function or the service. The one or more externalelectronic devices receiving the request may perform the at least partof the function or the service requested, or an additional function oran additional service related to the request, and transfer an outcome ofthe performing to the electronic device 101. The electronic device 101may provide the outcome, with or without further processing of theoutcome, as at least part of a reply to the request. To that end, acloud computing, distributed computing, or client-server computingtechnology may be used, for example.

FIG. 2 illustrates an example of a functional configuration of anelectronic device and an external electronic device according to variousembodiments. At least a part of the functional configuration of anelectronic device 201 and an external electronic device 202 in FIG. 2 isincluded in the functional configuration of the electronic device 101shown in FIG. 1 and the electronic device 102 shown in FIG. 1.

Referring to FIG. 2, the environment 200 includes the electronic device201 and the external electronic device 202.

The electronic device 201 is interlocked with the external electronicdevice 202 and is configured to provide contents related to virtualreality (VR) to the user. The electronic device 201 includes a memory210, a communication module 220, a display 230, a camera 240, or aprocessor 250.

In various embodiments, the memory 210 includes the memory 130 shown inFIG. 1, and the communication module 220 includes the communicationmodule 190 shown in FIG. 1, and display 230 includes the display device160 illustrated in FIG. 1, the camera 240 includes the camera 180illustrated in FIG. 1, and the processor 250 includes the processor 120illustrated in FIG. 1.

In various embodiments, the processor 250 (e.g., the processor 120) isoperatively or operably coupled with at least one of the memory 210, thecommunication module 220, and the display 230.

In various embodiments, the processor 250 acquires information about thelocation of the external electronic device 202 interlocked with theelectronic device 201 using the camera 240.

In one embodiment, the processor 250 acquires an image of the externalenvironment of the electronic device 201 using the camera 240. The imageof the external environment includes an image related to a plurality ofexternal objects including the external electronic device 202. In oneembodiment, the camera 240 is visually exposed through at least aportion of the first surface (the first side) of a housing for mountingthe components of the electronic device 201 (e.g., the memory 210, thecommunication module 220, the display 230, or the processor 250). Thehousing includes a second surface on an opposite side of the firstsurface, and the display 230 is visually exposed through at least aportion of the second surface.

In one embodiment, the camera 240 has a designated field of view. Thefield of view is referred to as various terms such as a visual field, aviewing angle, an angle of view, FOV (field of view), horizontal FOV(hFOV), and vertical FOV (vFOV). For example, the camera 240 acquires animage corresponding to an area of a predetermined size. The region ofthe predetermined size includes regions formed by a predetermined anglewith respect to the top, bottom, left, and right sides. For example,referring to FIG. 6, an area of the predetermined size corresponds to anarea included in the image 610. The user's field of view 401 correspondsto a region of a field of view (FOV) that is displayed on the display.The camera field of view 403 is determined based on the capability ofthe lenses included in the camera 240. For example, the camera 240includes a wide angle lens, and the camera field of view 403 has a largevalue angle corresponding to the wide angle lens. Referring to FIG. 4B,the camera field of view 403 corresponds to the areas 412, 414 and 416indicated by dotted lines.

In another embodiment, the camera 240 acquires an image for an area of apredefined size. Based on the field of view of the camera 240, the sizeof the area is different. For example, if the camera 240 corresponds toa wide lens camera, the size of the area increases. For example,referring to FIG. 4B, a camera field of view (FOV) corresponds to a userfield of view 401 and corresponds to a field of view 403 larger than theimage area 410 displayed on the display.

Hereinafter, the image 410 displayed in the display 230 includes imagescorresponding to a user's field of view. In various embodiments, thefield of view of the camera 240 is the same with the field of view of adisplayed area of the image 410 displayed on the display 230. However,the field of view of the camera 240 is not limited thereto, and asillustrated in FIG. 4B, it is smaller or larger than the user's field ofview.

In another embodiment, the processor 250 is linked or interlocked withthe external electronic device 202. For example, the interlockingincludes a connection created by a wired or a wireless way. Theinterlocking is performed by the communication module 220 included inthe electronic device 201.

In another embodiment, the processor 250 detects the external electronicdevice 202 by detecting a signal output from the external electronicdevice 202. For example, the external electronic device 202 furtherincludes a light emitting part. The light emitting part includes atleast one light emitting diode (LED). For example, the externalelectronic device 202 outputs a light corresponding to illuminance overor equal to a threshold range (e.g., a threshold value) through the LED.The processor 250 of the electronic device 201 identifies the externalelectronic device 202 by receiving the light corresponding toilluminance over or equal to the threshold value. In one embodiment,when the processor 250 detects a signal output from the LED of theexternal electronic device in the image acquired by the camera 240, itis determined that the external electronic device 202 is located withinthe field of view of the camera 240. For another example, if theprocessor 250 does not detect a signal output from the LED of theexternal electronic device in the image acquired by the camera 240, itis determined that the external electronic device 202 is not locatedwithin the field of view of the camera 240. That is, when the areacorresponding to the illuminance exceeding the threshold value is notdetected in the acquired image, the processor 250 determines that theexternal electronic device 202 deviates from the field of view of thecamera 240.

According to an embodiment, the LED included in the external electronicdevice 202 is turned on or off at predetermined time intervals. Theelectronic device 201 detects the external electronic device 202 byidentifying information on the predetermined time intervals for ON orOFF of the LED included in the external electronic device 202 byrecognizing that the external electronic device 202 is turned on or offat predetermined time intervals. For example, the external electronicdevice 202 controls the LED to repeat ON or OFF every 0.1 seconds. Theexternal electronic device 202 transmits periodic information forproviding the predetermined time intervals to the electronic device 201when the external electronic device 201 controls the LED to be turned onor off at predetermined time intervals. The electronic device 201receives the period information and obtains information about thepredetermined time interval for ON or OFF of the LED included in theexternal electronic device 202. The electronic device 201 detects theexternal electronic device 202 by identifying an LED that is turned onor off in the same manner as the predetermined time interval based onthe period information. In one embodiment, the predetermined timeinterval is randomly determined. For example, the predetermined timeinterval is determined based on the unique ID of the external electronicdevice 202. Since an ON or OFF time interval is determined based on theunique ID of the external electronic device 202 interlocked with theelectronic device 201, even if the electronic device 201 recognizes,through the camera 240, the LED output from another external electronicdevice (e.g., the VR controller of another user) interlocked withanother electronic device, the electronic device 201 accuratelyidentifies the LED output from the external device 202 interlocked withthe electronic device 201.

According to an embodiment, the external electronic device 202 measuresthe brightness of the external environment through an illuminance sensor(not shown) included in the external electronic device 202. When thebrightness of the external environment is greater than or equal to apredetermined value, the external electronic device 202 changes the LEDoutput intensity. For example, the external electronic device 202increases the LED output intensity in response to detecting that themeasured brightness for the external environment is greater than apredetermined threshold. The external electronic device 202 is moreeasily identified by the electronic device 201 by increasing the LEDoutput intensity. The external electronic device 202 turns the LED on oroff according to the predetermined time interval based on the increasedLED output intensity.

In another embodiment, the electronic device 201 acquires informationabout the location of the external electronic device 202. The locationinformation includes information determined based on an image obtainedfrom the camera 240 of the external electronic device 202. For example,the location information includes information about coordinates in whichthe external electronic device 202 is located in the acquired image. Thelocation information includes an x coordinate value, a y coordinatevalue, or a z coordinate value. Information about the coordinates isobtained by the location tracking module 252 included in the processor250.

In another embodiment, the processor 250 includes a location trackingmodule 252. The location tracking module 252 acquires information aboutthe location of the external electronic device 202 included in the imageacquired by the camera 240. For example, the location tracking module252 divides the acquired image into equally spaced coordinated in x-axisand y-axis, and obtains x-coordinates or y-coordinates corresponding tothe region in which the external electronic device 202 is detected.However, in various embodiments, the x-axis coordinates and the y-axiscoordinates are not limited to be obtained based on the division of thesame interval, and is also obtained based on techniques for obtaininginformation about multiple locations. For another example, the locationtracking module 252 acquires depth information based on the acquiredimage. The depth information is obtained based on an image operation forcalculating the depth information from a two-dimensional image.According to another embodiment, the location tracking module 252acquires depth information based on a difference value for two images.For example, the electronic device 201 includes two cameras, and the twocameras acquire images of the external electronic device 202,respectively. The two cameras are arranged side by side independentlywith a certain distance. The location tracking module 252 acquires thedepth information based on a difference between respective imagesobtained from the two cameras. The location tracking module 252 obtainsa z coordinate value according to the obtained depth information.

In various embodiments, the processor 250 receives information ontilting of the external electronic device 202 from the externalelectronic device 202. The processor 250 receives information on thetilting via the communication module 220.

According to an embodiment, the tilt information includes informationobtained by at least one sensor included in the external electronicdevice 202. For example, the at least one sensor includes a gyro sensor.The gyro sensor acquires sensor data (hereinafter, first sensor data)for tilting of the external electronic device 202. The first sensor dataincludes sensor data related to at least one of a pitch, a roll, or ayaw.

According to an embodiment, the tilt information is transmitted from theexternal electronic device 202 to the electronic device 201. Forexample, the tilt information is transmitted to the electronic device201 based on a state change of the external electronic device 202. Theexternal electronic device 202 transmits the tilt information to theelectronic device 201 in response to the turn on of power, on status,power on, or activation state of the external electronic device 202. Thetilt information is periodically transmitted from the externalelectronic device 202 to the electronic device 201. In one embodiment,the external electronic device 202 further includes a grip sensor (notshown). The grip sensor is configured to detect that the externalelectronic device 202 has been gripped by the user. The processor 295(e.g., the processor 250) receives a signal indicating that the externalelectronic device 202 has been gripped by the user from the grip sensor,and in response to receiving the indicated signal, the processor 295activates the gyro sensor. The processor 295 acquires information aboutthe tilt via the activated gyro sensor and transmits it to theelectronic device 201.

For another example, the external electronic device 202 transmitsinformation about the tilt to the electronic device 201 in response toreceiving a control signal from the electronic device 201. Theelectronic device 201 detects that interlocking with the externalelectronic device 202 is completed, and transmits a control signalrequesting the external electronic device 202 to transmit information onthe tilt in response to the detection. The control signal includes asignal indicating to activate the gyro sensor and/or the accelerationsensor. The external electronic device 202 activates the gyro sensor andacquires the first sensor data in response to the control signal. Theexternal electronic device 202 activates the acceleration sensor andacquires the second sensor data in response to the control signal.

In various embodiments, the processor 250 provides a graphic object(visual object) for representing the external electronic device 202together with multimedia content related to the virtual reality (VR).The graphic object refers to an object displayed on the display 230 toindicate user input in multimedia content related to virtual reality.The user input includes information about the movement of the externalelectronic device 202. The graphic object is referred to as variousterms such as a visual object, a graphical object, and a controllerobject. According to an embodiment, the graphic object includes imagesof various shapes. For example, the graphic object includes an image ofthe same or similar shape to that of the external electronic device 202.For another example, the external electronic device 202 includes animage determined based on the type of multimedia content. The imagedetermined based on the type of content includes images for variousobjects including a tennis racket, steering wheel, knife or sword, andthe like.

In one embodiment, the processor 250 displays multimedia content relatedto the virtual reality. The multimedia content includes at least one ofa three-dimensional or an omni-directional image for providing a virtualreality service to a user.

In another embodiment, the processor 250 displays the graphic object forrepresenting the external electronic device 202 on the display 230. Forexample, the graphic object is displayed to be superimposed with themultimedia content related to the VR. The processor 250 rendersmultimedia content related to the VR at a first time point and rendersthe graphic object. The graphic object is displayed superimposed on themultimedia content.

In another embodiment, the graphic object is identified based oninformation on the location of the external electronic device 250 orinformation on the tilting of the external electronic device 202. Forexample, the processor 250 obtains information about the location of theexternal electronic device 202, and identifies a location to display thegraphic object. For example, the processor 250 determines to display thegraphic object in three dimensions at the identified display locationbased on the information about the tilting. Since the processor 250identifies only the location of the external electronic device 202 basedon the illuminance value in the image acquired via the camera 240, thetilt information is needed as the information for rendering in threedimensions. For another example, the processor 250 acquires informationabout the tilt based on a pattern or a marker of the external electronicdevice 202 included in the image acquired via the camera 240. In oneembodiment, the external electronic device 202 includes at least onemarker. Since the at least one marker is located in a fixed area of theexternal electronic device 202, when the external electronic device 202is tilted, the at least one marker is also tilted in the same way. Theprocessor 250 acquires an image of the at least one tilted marker in animage acquired via the camera 240. In another embodiment, the externalelectronic device 202 outputs a light of a predetermined pattern throughthe LED. For example, when the external electronic device 202 isinclined to form a perpendicular angle to the ground surface, theexternal electronic device 202 turns the LED on or off at 0.1 secondintervals. For another example, when the external electronic device 202is inclined side by side with the ground surface, the externalelectronic device 202 turns the LED on or off at 0.5 second intervals.The external electronic device 202 indirectly provides information aboutthe tilt of the external electronic device 202 by transmitting theinformation on the predetermined pattern to the electronic device 201.

In yet another embodiment, the graphic object is determined based onmultimedia content related to the VR. The multimedia content related tothe VR includes a tennis match, a war game, a car game, and the like.For example, when the multimedia content corresponds to a tennis match,the graphic object corresponds to an image including the shape of atennis racket. For another example, when the multimedia contentcorresponds to the war game, the graphic object corresponds to an imageincluding the shape of a sword or a knife. As another example, when themultimedia content corresponds to the car game, the graphic objectcorresponds to an image including a shape of a steering wheel.

The processor 250 determines a point corresponding to the value of thex-coordinate and/or the y-coordinate on the display 230 by referring toa value of the x-coordinate and/or y-coordinate among the information onthe location. The processor 250 determines a size of the determinedpoint by referring to a value for z coordinates among the information onthe location. For example, when the value for the z coordinate is large(e.g., the external electronic device 202 is far from the user), thedetermined point for the perspective effect is displayed small. Foranother example, when the value for the z coordinate is small (e.g., theexternal electronic device 202 is located close to the user), thedetermined point for the perspective effect is largely displayed. Theprocessor 250 determines a three-dimensional display of the graphicobject based on the tilt information. For example, the processor 250determines the tilted direction or angle in three dimensions byreferring to the gyro sensor data indicating at least one of a pitch, aroll, or a yaw among the information on the tilt. The processor 250displays the graphic object inclined according to the tilted directionor angle in 3D at a location determined according to the x coordinate, ycoordinate, or z coordinate. For example, referring to FIG. 6, theprocessor 250 displays the display image 620 via the display 230. Thedisplay image 620 includes a background image or the graphic object 602related to the multimedia content. The processor 250 displays thegraphic object 602 on an area corresponding to the location of theexternal electronic device 102 identified based on the camera image 610.The processor 250 displays the graphic object 602 to have the same tiltbased on the information about the tilt of the external electronicdevice 202 received from the external electronic device 202. The image620 displayed via the display 230 includes images corresponding to auser's field of view. In various embodiments, the field of view of thecamera 240 is the same as the field of view of the display area of theimage 620 displayed via the display 230.

In various embodiments, the processor 250 identifies that the externalelectronic device 202 deviates from the field of view of the camera 240.For example, the processor 250 identifies whether the externalelectronic device 202 deviates from the field of view of the camera 240based on an image acquired by the camera 240. The processor 250determines that the external electronic device 202 has deviated from thefield of view of the camera 240 when an area corresponding toilluminance of a threshold value or higher is not detected in theacquired image. As another example, if the processor 250 detects an areacorresponding to the illuminance of the threshold value or higher in animage acquired by the camera 240, the processor 250 determines that theexternal electronic device 202 is inside the field of view of the camera240. For another example, referring to FIG. 4A, the user grips each ofthe external electronic devices 202-1 and 202-2 with both hands whilewearing the electronic device 201. Each of the external electronicdevices 202-1 and 202-2 corresponds to the external electronic device202 shown in FIG. 2. Referring to FIG. 4A, when a user of the electronicdevice 201 faces an arm forward (e.g., indicated by a solid line), theexternal electronic devices 202-1 and 202-2 are within a view of angles(401) of the electronic device 201, and the electronic device 201acquires an image including external electronic devices 202-1 and 202-2.As the processor 250 detects an area corresponding to the illuminance ofa threshold value or higher, which is emitted from the externalelectronic devices 202-1 and 202-2, and thereby the processor 250identifies that the external electronic devices 202-1 and 202-2 islocated inside the field of view of the electronic device 201. Referringto FIG. 4A, when a user of the electronic device 201 points the arm tothe side (e.g., as in the example of an arm indicated by a dotted line),the external electronic devices 202-1 and 202-2 deviates from theviewing angle 401 of electronic devices 201, and the electronic device201 does not acquire an image including external electronic devices202-1 and 202-2. The electronic device 201 does not detect an areacorresponding to the illuminance above a threshold value or higher,emitted from the external electronic devices 202-1 and 202-2. Theelectronic device 201 identifies that the external electronic devices202-1 and 202-2 deviates from field of view of the camera 240 or are outof field of view of the camera 240.

According to an embodiment, the processor 250 stores information on thelocation of the external electronic device 202 corresponding to thedeviation time point. For example, the processor 250 identifies thelocation of the external electronic device 202 in response toidentifying the deviation. The identified location is located on one ofthe boundary surfaces of the image acquired by the camera 240. Theprocessor 250 stores information on the identified location as secondlocation information. The second location information is obtained by themost recent image including the external electronic device 202 among aplurality of images acquired by the camera 240. When the image obtainedby the camera 240 is divided into the x-axis along the horizontal axisand the y-axis along the vertical axis, the second location informationincludes information about one of the points having only one of thex-coordinate value or the y-coordinate value.

In various embodiments, the processor 250 transmits a request, inresponse to the above identification, to the external electronic device202 to transmit information about a change in the location of theexternal electronic device 202. When the external electronic device 202deviates from the field of view of the camera 240, the camera 240 doesnot acquire an image including the external electronic device 202, andaccordingly, the processor 250 does not acquire information about thelocation of the external electronic device 202. Accordingly, when theprocessor 250 detects the deviation of the external electronic device202, the processor 250 transmits a request for information about thechange in the location of the external electronic device 202 to theexternal electronic device 202.

In one embodiment, the information about the change in location includesinformation obtained by at least one sensor of the external electronicdevice 202. For example, the at least one sensor included in theexternal electronic device 202 includes a first sensor 292 or a secondsensor 294. The first sensor 292 corresponds to a gyro sensor, and thesecond sensor 294 corresponds to an acceleration sensor. The firstsensor 292 acquires information about the tilting of the externalelectronic device 202, and the second sensor 294 acquires informationabout the change in speed of the external electronic device 202. Theexternal electronic device 202 transmits information obtained from thefirst sensor 292 or the second sensor 294 to the electronic device 201in response to a request for information about the change in thelocation. That is, the external electronic device 202 transmits only theinformation on the tilt by the first sensor 292 to the electronic device201, and in response to the request, the information on the tilt by thefirst sensor 292 and information on the speed change by the secondsensor 294 are transmitted together. The transmitted tilt information orinformation on the speed change is used by the processor 250 of theelectronic device 201 to predict location of the external electronicdevice 202 at a time point after the deviation of the externalelectronic device 202.

In another embodiment, the information about the change in locationincludes data indicating the amount of change in the location of theexternal electronic device 202. For example, the data indicating theamount of location change includes vector information. Data indicatingthe amount of location change is obtained by the external electronicdevice 202. The external electronic device 202 acquires the vectorinformation based on the information obtained by the first sensor 292 orthe second sensor 294. The external electronic device 202 transmits thevector information obtained based on the sensor data, instead oftransmitting the sensor data acquired by the first sensor 292 or thesecond sensor 294.

In one embodiment, the processor 295 predicts a moving direction of theexternal electronic device 202 based on information on the tilt. Forexample, the processor 295 predicts movement of the external electronicdevice 202 in a direction corresponding to the rotation direction. Thisis because, while the user holds the external electronic device 202 byhand, when the data by the gyro sensor is detected, the expected valuefor the case where the data is detected by the rotational movement inplace is very low. Accordingly, when the processor 295 acquires dataindicating that the external electronic device 202 has rotated in the x+direction, the processor 295 predicts that the external electronicdevice 202 has moved in the x+ direction. For another example, theprocessor 295 stores information about a change in the location forrepetitive operation. In another embodiment, the processor 295 comparesdata obtained from the first sensor 292 or the second sensor 294 withthe stored information. When the obtained data and the storedinformation match, the processor 295 determines that the user grippingthe external electronic device 202 is performing a repetitive operation.The processor 250 displays a graphic object on the display 230 based onthe determined repetitive operation.

In another embodiment, a request signal requesting to transmitinformation about the change in location includes a control signalindicating to activate the second sensor 294 of the external electronicdevice 202. The external electronic device 202 activates the secondsensor 294 in response to receiving the control signal included in therequest signal, and obtains information about the speed change.According to an embodiment, when the processor 250 identifies theexternal electronic device 202 based on the image acquired by the camera240 (e.g., the external electronic device 202 is within the camera fieldof view 403), the control signal instructing to activate the firstsensor 292 (e.g., a gyro sensor) is transmitted to the externalelectronic device 202. The processor 250 receives information about adirection and/or angle in which the external electronic device 202 istilted from the first sensor 292. The processor 250 renders the externalelectronic device 202 according to the tilted direction and/or angle onthe location of the external electronic device 202 in the image acquiredby the camera 240. According to another embodiment, when the processor250 does not identify the external electronic device 202 based on theimage acquired by the camera 240 (e.g., deviation of the externalelectronic device 202 from the camera field of view 403 is detected),control signals instructing to activate both the first sensor 292 (e.g.,gyro sensor) and/or the second sensor 294 (e.g., acceleration sensor)are transmitted to the external electronic device 202. The processor 250receives information on a speed change of the external electronic device202 outside the camera field of view 403 from the second sensor 294. Theprocessor 250 acquires a location to which the external electronicdevice 202 is predicted to have moved based on the information on thespeed change. The processor 250 receives information on a tilteddirection and/or a tilted angle of the external electronic device 202from the gyro sensor. The processor 250 renders a graphic objectcorresponding to the external electronic device 202 on a location wherethe external electronic device 202 is predicted to have moved. Theprocessor 250 displays the graphic object based on the tilted directionand/or the tilted angle.

According to various embodiments, the processor 250 changes a display ofthe graphic object based at least on information on the location of theexternal electronic device 202 at a deviation time obtained using thecamera 240, information on the tilt of the external electronic device202 received from the external electronic device 202 or information on achange in the location of the external electronic device 202.

According to an embodiment, the processor 250 predicts the location ofthe external electronic device 202 corresponding to the current timepoint. For example, the processor 250 generates information about thechange in location of the external electronic device 202 based on theinformation on the tilt or the speed change of the external electronicdevice 202 received from the external electronic device 202. Theinformation about the generated location change includes vectorinformation. The vector information includes information on displacementof the external electronic device 202. For another example, theprocessor 250 receives information on the change of the location fromthe external electronic device 202. The information about the change oflocation received from the external electronic device 202 corresponds toinformation obtained by the processor 295 of the external electronicdevice 202 based on the information about the tilt or the speed change.The processor 250 predicts the current location of the externalelectronic device 202 based on information on the change in locationand/or information on the location of the external electronic device 202at the deviation time point. For example, the processor 250 determinesan area where the external electronic device 202 deviates as a startingpoint and, identifies change of location according to the movement ofthe external electronic device 202 after deviation from the camera'sfield of view based on the starting point and/or vector information. Theprocessor 250 displays a visual effect and/or a graphic object (e.g.,arrow) separate from the visual effect for indicating the direction towhich the external electronic device is located based on the predictedlocation of the external electronic device 202.

In various embodiments, the starting point and/or the vector informationis changed based on the movement of the electronic device 201. Theprocessor 250 acquires information about the movement of the electronicdevice 201 after the external electronic device 202 is out of the camerafield of view 403. Information about the movement of the electronicdevice 201 is obtained by a sensor module (e.g., the sensor module 176in FIG. 1) equipped in the electronic device 201. In one embodiment,after the external electronic device 202 deviates from the camera fieldof view 403, when the electronic device 201 rotates in the leftdirection (for example, wearing a VR device and rotating the head to theleft), the starting point is changed based on the movement of theelectronic device 201. For example, the starting point is moved to theleft direction from the region at the time when the external electronicdevice 202 is deviated, by reflecting the leftward rotation of theelectronic device 201. In another embodiment, if the electronic device201 rotates in the left direction after the external electronic device202 deviates from the camera field of view 403, vector information ischanged based on the motion of the electronic device 201. For example,the vector information further includes information for indicating anadditional location change to the right direction by reflecting therotation in the left direction of the electronic device 201.

According to an embodiment, the processor 250 displays a part of thefirst graphic object. The first graphic object refers to a graphicobject determined based on the multimedia content. For example, when thefirst graphic object corresponds to a sword, the processor 250 displaysa part of the first graphic object with reference to the length of thesword. That is, when the external electronic device 202 deviates fromthe field of view of the camera 240, but does not move beyond the lengthof the sword, the processor 250 displays a part of the first graphicobject. The processor 250 predicts the current location using vectorinformation for the location change and the deviation location of theexternal electronic device 202, and when the distance between thepredicted current location and the camera 240 is less than the length ofthe sword, a part of sword (for example, a blade) is displayed.

According to an embodiment, the processor 250 display the second graphicobject which is different from the first graphic object by omitting thedisplay of the first graphic object. The second graphic objectcorresponds to an object for indicating the user that the externalelectronic device 202 is located in an external area of the imagedisplayed on the display 230. The second graphic object is an objectdifferent from the first graphic object. The second graphic objectincludes an image of the same or similar shape to that of the externalelectronic device 202. For another example, the second graphic objectincludes an arrow-shaped image for indicating that the externalelectronic device 202 is located outside the field of view of the camera240. The processor 250, along with the second graphic object, furtherdisplays a guide phrase indicating that the external electronic device202 is located outside the field of view of the camera 240. For example,when the distance between the predicted location of the externalelectronic device 202 and the camera 240 exceeds the length of the firstgraphic object, the processor 250 does not display the first graphicobject. The processor 250 displays the second graphic object forindicating the current location of the external electronic device 202because the first graphic object is not displayed.

The external electronic device 202 is configured to interlock with theelectronic device 201 and provide user input in content related tovirtual reality. The external electronic device 202 includes a memory260, a communication module 270, a user input module 280, a sensormodule 290 or a processor 295.

In various embodiments, the memory 260 includes the memory 130 shown inFIG. 1, and the communication module 220 includes the communicationmodule 190 shown in FIG. 1, and a user The input module 280 includes theinput device 150 shown in FIG. 1, and the sensor module 290 includes thesensor module 176 shown in FIG. 1.

In various embodiments, the user input module 280 receives a user inputfor controlling the electronic device 201 interlocked with the externalelectronic device 202. The user input module 280 is visually exposedthrough a part of the housing (not shown) of the external electronicdevice 202. In various embodiments, the user input module 280 includes atouch pad for receiving a user's touch input, a physical button capableof receiving physical pressure, or a physical key.

In various embodiments, the processor 295 is operatively or operablycoupled with at least one of the memory 260, the communication module270, the user input module 280, or the sensor module 290.

In various embodiments, the sensor module 290 includes a first sensor292 or a second sensor 294. The first sensor 292 corresponds to a gyrosensor, and the second sensor 294 corresponds to an acceleration sensor.

In various embodiments, the processor 295 transmits information aboutthe tilting of the external electronic device 202 to the electronicdevice 201. For example, the processor 295 acquires information aboutthe tilting via at least one sensor. The at least one sensor includes agyro sensor. The tilt information includes information for indicating a3D angle change of the external electronic device 202. The tiltinformation includes data on a change amount of a pitch, a roll, and ayaw.

In various embodiments, the processor 295 transmits information about achange in the location of the external electronic device 202 to theelectronic device 201. The information on the change in locationincludes information on the tilt or information on the speed changeobtained by the first sensor 292 or the second sensor 294 of theexternal electronic device 202. For another example, the informationabout the change in location includes information indicating the amountof change in location obtained based on the information about the tiltor the speed change. The information indicating the location changeamount corresponds to the vector information.

In one embodiment, the processor 295 adaptively transmits informationabout the change in the location to the electronic device 201. Forexample, the processor 295 transmits information on the change in thelocation to the electronic device 201 at predefined time intervals. Thepredefined time interval is changed based at least on the remainingbattery power of the external electronic device 202 and the quality ofwireless communication between the external electronic device 202 andthe electronic device 201. For example, when the charge level of thebattery of the external electronic device 202 is high, the predefinedtime interval is reduced to reflect the movement of the externalelectronic device 202 in real time. For another example, when the chargelevel of the battery of the external electronic device 202 is low, inorder to increase the driving time of the external electronic device 202and reduce power consumption, the predefined time interval is increased.For example, when the quality of the wireless communication is poor, theprocessor 295 reduces the predefined time interval to ensure reliabilityof data transmission. For another example, the processor 295 increasesthe predefined time interval when the quality of the wirelesscommunication is good.

In another embodiment, the processor 295 transmits information about achange in the location along with different information transmitted tothe electronic device 201. For example, the processor 295 receives auser's physical key input and transmits information on the change of thelocation along with the received input.

Although not shown, the external electronic device 202 further includesa light emitting part. For example, the light emitting part activatesthe light emitting part in response to detecting an on status of theexternal electronic device 202. For another example, the externalelectronic device 202 generates a light in response to receiving asignal instructing to emit light from the electronic device 201. Theelectronic device 201 is configured to acquire information on thelocation of the external electronic device 202 by capturing the lightemitted from the light emitting part through the camera 240. In variousembodiments, the light emitting part includes a light source of a lightemitting diode (LED).

FIG. 3 illustrates an example of a functional configuration of anelectronic device, a first external electronic device, and a secondexternal electronic device according to various embodiments. Thefunctional configuration of the second external electronic device 303illustrated in FIG. 3 is included in the functional configuration of theexternal electronic device 202 illustrated in FIG. 2. The functionalconfiguration of the electronic device 301 illustrated in FIG. 3 isincluded in the functional configuration of the electronic device 201illustrated in FIG. 2.

Referring to FIG. 3, the first external electronic device 302 includes amemory 321, a communication module 322, a display 323, a camera 324 or aprocessor 325.

In various embodiments, the memory 321 includes the memory 210 shown inFIG. 2, and the communication module 322 includes the communicationmodule 220 shown in FIG. 2, The display 323 includes the display 230shown in FIG. 2, the camera 324 includes the camera 240 shown in FIG. 2,and the processor 325 includes the processor 250 shown in FIG. 2.

In various embodiments, the processor 325 is operatively coupled with atleast one of the memory 321, the communication module 322, the display323, and the camera 324.

In various embodiments, the processor 325 acquires information about thelocation of the second external electronic device 303. For example, whenthe second external electronic device 303 is located inside the field ofview of the camera 324, the processor 325 acquires the image includingthe second external electronic device 303 via the camera 324. Accordingto one embodiment, the processor 325 further includes a locationtracking module. The location tracking module acquires information aboutthe location of the second external electronic device 303 within theacquired image. The location tracking module corresponds to the locationtracking module 252 of the processor 250 illustrated in FIG. 2. Inanother embodiments, the information about the location of the secondexternal electronic device 303 is obtained by the electronic device 301.The processor 325 acquires an image including the second externalelectronic device 303 through the camera 324 and transmits the acquiredimage to the electronic device 301. The electronic device 301 obtainsthe information about the location of the second external electronicdevice 303 within the image including the second external electronicdevice 303 received from the first external electronic device 302.

In various embodiments, the processor 325 transmits information aboutthe location corresponding to the deviation time point of the secondexternal electronic device 303 to the electronic device 301. Forexample, the processor 325 determines whether the second externalelectronic device 303 has deviated from the field of view of the camera324 based on the image acquired by the camera 324. The processor 325determines, when detecting an area corresponding to illuminance of athreshold value or more in the acquired image, that the second externalelectronic device 303 is located in the detected area. For anotherexample, the processor 325 determines, when an area corresponding toilluminance of the threshold or more is not detected in the acquiredimage, that the second external electronic device 303 deviates from thefield of view of the camera 324. The processor 325 temporarily stores animage at a time point immediately before the time point when an areacorresponding to the illuminance above the threshold or more is notdetected in a buffer or the like. Accordingly, when the deviation isidentified, the processor 325 identifies an image including the secondexternal electronic device 303 among a plurality of images stored in abuffer or the like. The processor 325 acquires location information ofthe second external electronic device 303 at the deviation time from theidentified image using the location tracking module.

In various embodiments, the processor 325 transmits a signal indicatingthe deviation to the electronic device 301. For example, the processor325 transmits a signal indicating the deviation to the electronic device301 in response to identifying the deviation. Since the electronicdevice 301 does not have a camera, it does not identify the deviation ofthe second external electronic device 303. Accordingly, the processor325 transmits a signal indicating the deviation to the electronic device301. In various embodiments, the processor 325 transmits informationabout the location of the second external electronic device 303corresponding to the deviation time point, together with the abovesignal to the electronic device 301.

In various embodiments, the processor 325 displays an image related tothe multimedia content or a graphic object corresponding to the secondexternal electronic device 303 on the display 323. For example, theprocessor 325 receives data for the display from the electronic device301. The processor 325 receives location information by which the secondexternal electronic device 303 is predicted to be located from theelectronic device 301, and displays a graphic object based on thepredicted location information. The predicted location information isobtained by the electronic device 301. The predicted locationinformation is obtained on the basis of the information about a changein the location transmitted from the second external electronic device303 to the electronic device 301 or the information about the locationof the second external electronic device 303 at a deviation time pointtransmitted from the first external electronic device 302 to theelectronic device 301.

In various embodiments, the electronic device 301 generates a graphicobject to be output on the display 323 of the first external electronicdevice 302. The graphic object includes multimedia content related tothe VR (e.g., VR operation) and/or a graphic object corresponding to theexternal electronic device 303. For example, the processor 313 of theelectronic device 301 receives information about the location of thesecond external electronic device 303 obtained from the first externalelectronic device 302 and/or information on the location of the secondexternal electronic device 303 obtained from the second externalelectronic device 303, and generates a graphic object based on thereceived location information of the second external electronic device303. The graphic object is transmitted to the first external electronicdevice 302 and output via the display 323.

FIG. 4A illustrates an example of a situation in which an externalelectronic device deviates from a camera's field of view according tovarious embodiments. The electronic device 201 illustrated in FIG. 4Acorresponds to the electronic device 201 illustrated in FIG. 2. Theexternal electronic devices 202-1 and 202-2 shown in FIG. 4A correspondsto the external electronic device 202 shown in FIG. 2.

Referring to FIG. 4A, the electronic device 201 corresponds to a statein which the electronic device 201 is worn on the user's body. Theelectronic device 201 includes a display 230 on one surface of a housing(not shown). The display 230 is arranged to face the user's eye in theworn state of the electronic device 201. The electronic device 201includes a camera 240 on a surface opposite to one surface of thehousing. The camera 240 is disposed on the opposite surface to acquirean image of an external environment in a direction corresponding to auser's gaze.

According to various embodiments, the external electronic device 202corresponds to a state held by a user's hand. The external electronicdevice 202 includes two devices 202-1 and 202-2 (e.g., a controller) sothat they are gripped by the user's left and right hands, respectively.

According to various embodiments, the camera 240 included in theelectronic device 201 acquires an image of a field of view correspondingto a certain angle 401 (e.g., a viewing angle of the camera 240). Forexample, the camera 240 obtains an image of the external electronicdevice 202 when the external electronic device 202 is located within theangle 401 (e.g., when the user extends the arm in front). In oneembodiment, the image of the external electronic device 202 includes anexternal image of the external electronic device 202 or an image of thehousing. According to another embodiment, the image of the externalelectronic device 202 includes an LED, a pattern, or a marker. Forexample, the external electronic device 202 additionally includes an LEDseparately from the housing of the external electronic device in orderto be easily recognized by the processor 250. Instead of recognizing theexternal electronic device 202 by processing an image of the housing ofthe external electronic device 202, the processor 250 identifies animage of the external electronic device 202 within the image obtained bythe camera 240 by recognizing an LED emitting a brightness greater thanor equal to a threshold illuminance value. For another example, theexternal electronic device 202 includes at least one marker. Each of theat least one marker includes markers in a distinguishable pattern. Theprocessor 250 identifies the at least one marker from the image acquiredby the camera 240 and identifies an external electronic device 202within the acquired image. For another example, when the externalelectronic device 202 deviates from the angle 401 (e.g., when a userspreads his or her arms out to both sides), the external electronicdevice 202 deviates from a field of view of the camera 240 and thecamera 240 does not acquire an image 413 for the external electronicdevice 202.

FIG. 4B illustrates an example for comparing a field of viewcorresponding to a camera of an electronic device and a field of viewcorresponding to a user's gaze according to various embodiments.Referring to FIG. 4B, a size of an area displayed on the display 230 andan area of an image acquired through the camera 240 is different.According to an embodiment, the field of view of the camera 240 includesa larger area than the user's field of view (e.g., an area displayed onthe display 230). For example, referring to the image (a) illustrated inFIG. 4B, an area 410 indicated by a solid line corresponds to an areadisplayed on the display 230. The area 412 indicated by a dotted lineindicates the size of the area that the camera 240 acquires.Accordingly, the processor 250 acquires an image including the externalelectronic device 202 when the external electronic device 202 is locatedwithin the field of view of the camera 240 even if the externalelectronic device 202 is out of the user's field of view. The processor250 obtains information on the location of the external electronicdevice 202 based on the image including the external electronic device202. Accordingly, the processor 250 displays the graphic object 411corresponding to the external electronic device 202 based on theobtained location information.

According to another embodiment, the field of view of the camera 240 isthe same as that of the user. For example, referring to the image (b)illustrated in FIG. 4B, the size of the area 410 indicated by a solidline is the same as or similar to the size of the area 414 indicated bya dotted line. Accordingly, when the external electronic device 202 isout of the user's field of view, the processor 250 does not acquire animage of the external electronic device 202 because it is out of thefield of view of the camera 240. When the external electronic device 202is located inside the user's field of view, the processor 250 acquiresan image of the external electronic device 202 because it is locatedinside the field of view of the camera 240. Accordingly, the processor250 obtains information on the location of the external electronicdevice 202 based on the image, and displays a graphic object 411corresponding to the external electronic device 202.

According to another embodiment, the field of view of the camera 240includes an area smaller than that of the user. For example, referringto the image (c) illustrated in FIG. 4B, the size of the area 416indicated by a dotted line is smaller than the area 410 indicated by asolid line. Accordingly, the processor 250 does not acquire an imageincluding the external electronic device 202 when the externalelectronic device 202 is located outside the field of view of the camera240 even though the external electronic device 202 is located inside theuser's field of view. The processor 250 transmits a request signal tothe external electronic device 202 to transmit information about achange in the location of the external electronic device 202. Theprocessor 250 acquires the location of the external electronic device202 based on the information on the change in the location, and displaysa graphic object 411 corresponding to the external electronic device 202on the acquired location.

FIG. 5A illustrates an operation of an electronic device according tovarious embodiments. Such an operation is performed by the electronicdevice 201 illustrated in FIG. 2 or the processor 250 included in theelectronic device 201.

Referring to FIG. 5A, in operation 501, the processor 250 identifies anexternal electronic device among one or more external objects includedin a designated field of view using a camera. The processor 250 acquiresan image of the surrounding environment of the electronic device 201through the camera. The image includes a plurality of objects locatedaround the electronic device 201. The processor 250 identifies theexternal electronic device 202 among the plurality of objects. Forexample, the external electronic device 202 further includes a lightemitting part. The processor 250 identifies the external electronicdevice 202 by identifying a light source exceeding a predeterminedilluminance in the image acquired through the camera.

According to an embodiment, the external electronic device 202 adjuststhe brightness of the LED based on the ambient brightness. The externalelectronic device 202 further includes an illuminance sensor (notshown), and acquire a value of the lux (illuminance) of the ambientbrightness of the external electronic device 202 using the illuminancesensor (not shown). The external electronic device 202 adjusts thebrightness of the light source based on the value of the lux of theambient brightness. For example, the external electronic device 202outputs light having a first brightness corresponding to the firstsection in response to identifying that the value of the lux of theambient brightness is included in the first section. For anotherexample, in response to identifying that the value of the lux of theambient brightness is included in the second section, light of thesecond brightness corresponding to the second section is output. Thefirst brightness corresponds to an illuminance value required to beseparately identified from the ambient brightness of the first section.The second brightness corresponds to an illuminance value required to beseparately identified from the ambient brightness of the second section.

In operation 503, the processor 250 displays a graphic objectcorresponding to the external electronic device on the display based onthe first location information of the external electronic device 202.The processor 250 displays the graphic object on the identified locationof the external electronic device 202. The graphic object is determinedbased on multimedia content. For example, the graphic object includesimages of various shapes, such as a knife, a tennis racket, or a carwheel.

In operation 505, the processor 250, if the external electronic device202 is out of the designated field of view, displays the graphic objecton the display based on the second location information confirmedthrough the camera before the external electronic device 202 is out ofthe designated field of view and information related to movement of theexternal electronic device received from the external electronic devicethrough the communication circuitry after the external electronic device202 is out of the designated field of view. According to an embodiment,the processor 250 stores the second location information at a time pointwhen the external electronic device 202 deviates from the designatedfield of view. The second location information is located at one pointof the boundary surface of the image displayed on the display 210.According to an embodiment, the processor 250 receives informationrelated to the movement of the external electronic device 202 from theexternal electronic device 202 from a time point at which the externalelectronic device 202 is out of the designated field of view. Themovement related information includes data obtained by a gyro sensorand/or data obtained by an acceleration sensor. The processor 250generates a vector value for movement of the external electronic devicebased on the information related to the movement. The vector value isobtained by determining the size and/or direction of the vectoraccording to information related to the movement, with a region wherethe external electronic device 202 deviates as a starting point.

FIG. 5B illustrates an operation of an electronic device according tovarious embodiments. Such an operation is performed by the electronicdevice 201 illustrated in FIG. 2 or the processor 250 included in theelectronic device 201.

In operation 511, the processor 250 obtains information about thelocation of the external electronic device 202 interlocked with theelectronic device 201. Information on the location of the externalelectronic device 202 refers to information on coordinate valuesidentified by the external electronic device 202 in an image acquired bythe camera 240. In one embodiment, the processor 250 acquires an imagevia the camera 240. The processor 250 obtains an x coordinate valueand/or a y coordinate value based on the image. For example, byanalyzing the image, the processor 250 identifies a coordinate valuecorresponding to an area in which the external electronic device 202 islocated in the image. In another embodiment, the processor 250 obtainsdepth information based on image analysis of the image. The processor250 obtains a z coordinate value based on the obtained depth informationand/or the obtained x coordinate value and y coordinate value.

In operation 513, the processor 250 receives tilt information about thetilting of the external electronic device 202. The tilt informationincludes sensor data obtained by the first sensor 292 included in theexternal electronic device 202. For example, the tilt informationincludes information about at least one change amount of pitch, roll, oryaw. According to an embodiment of the present disclosure, the tiltinformation is configured to be automatically transmitted from theexternal electronic device 202 to the electronic device 201 in responseto an on-status or an activation state of the external electronic device202. According to another embodiment, the tilt information is configuredto be transmitted to the electronic device 201 in response to therequest from the electronic device 201.

In operation 515, the processor 250 provides a graphic object forrepresenting the external electronic device 202 together with multimediacontent. The processor 250 uses the obtained information on the tilt ofthe external electronic device 202 and/or the location of the externalelectronic device 202 obtained by the camera 240, thereby allowing thegraphic object to be displayed. For example, the processor 250 obtainscoordinate values for x, y, and z from the camera 240, and receivesinformation on tilt from the external electronic device 202, anddetermines the shape of the external electronic device 202 in a threedimensional space. The processor 250 displays a graphic object tocorrespond to the determined shape of the external electronic device202. For example, the graphic object is displayed with the same tilt onthe same coordinates as the external electronic device 202. According tovarious embodiments, the graphic object is determined based onmultimedia content. For example, when the multimedia content istennis-related content, the graphic object includes a shape of a tennisracket. For another example, when the multimedia content is contentrelated to a combat game, the graphic object includes the shape of aknife or sword.

In operation 517, the processor 250 identifies that the externalelectronic device 202 is out of field of view of the camera. Theprocessor 250 determines that the external electronic device 202 doesnot exist based on the image acquired by the camera 240. For example,when the image acquired by the camera 240 does not include an image oflight emitted from a light emitting part (not shown) of the externalelectronic device 202, the processor 250 identifies the deviation of theelectronic device 202. For another example, when a region brighter thana predetermined illuminance is not detected in the acquired image, theprocessor 250 determines that the external electronic device 202 hasdeviated. According to an embodiment, the processor 250 storesinformation on the location of the external electronic device 202 at thedeviation time point. For example, the processor 250 stores informationon the location where the external electronic device 202 deviates inresponse to detecting the deviation.

In operation 519, the processor 250 requests to transmit informationabout a change in the location of the external electronic device 202.When the external electronic device 202 deviates from the field of viewof the camera 240, the processor 250 does not acquire information aboutthe location of the external electronic device 202. The processor 250transmits a signal requesting the external electronic device 202 totransmit information about a change in the location of the externalelectronic device 202. According to various embodiments, the requestsignal includes a control signal instructing to activate the secondsensor 294 of the external electronic device 202. For example, when theexternal electronic device 202 is included in the field of view of thecamera 240, the external electronic device 202 activates only the firstsensor 292, and information about tilting obtained from the first sensor292 is transmitted to the electronic device 201. For another example,when the external electronic device 202 deviates from the field of viewof the camera 240, the external electronic device 202 receives thecontrol signal from the electronic device 201, and activates the secondsensor 294. The external electronic device 202 transmits information onthe tilt obtained by the first sensor 292 and/or information on a changein speed of the external electronic device 202 obtained by the secondsensor 294 to the electronic device 201. For another example, the secondsensor 294 transmits information on the speed change to the electronicdevice 201 regardless of receiving a control signal indicating theactivation of the second sensor 294 from the electronic device 201. Theprocessor 295 of the external electronic device 202 confirms (e.g.,detects) information related to driving the external electronic device202 (e.g., power-on), and in response to the confirmation, activates thesecond sensor 294. In addition, the processor 295 further includes agrip sensor (not shown), and the grip sensor (not shown) detects thatthe external electronic device 202 is held by the user. The processor295 receives a sensor value from the grip sensor (not shown), andactivates the second sensor 294 in response to the reception of thesensor value. When the second sensor 294 is activated in response to apower-on or receiving the sensor value from the grip sensor (not shown)of the external electronic device 202, information about the tilt of theexternal electronic device 202 acquired by the first sensor 292 andinformation about the speed change of the external electronic device 202obtained by the second sensor 294 are transmitted to the electronicdevice 201 together. According to various embodiments, the processor 250obtains a vector value for a change in the location of the externalelectronic device 202. The processor 250 receives information on thechange of the location from the external electronic device 202, and thereceived information includes the data acquired by the first sensor 292and/or the second sensor 294. The processor 250 obtains a vector valuebased on the data on the tilting of the external electronic device 202obtained by the first sensor 292 and/or the data on the speed change ofthe external electronic device 202 obtained by the second sensor 294.The vector value includes information indicating how far and in whatdirection the external electronic device 202 has moved from locationinformation corresponding to a deviation time point from the field ofview of the camera 290.

In various embodiments, the processor 250 determines information about achange in the location of the external electronic device 202 based oninformation about the change in the location of the electronic device201. For example, the electronic device 201 includes a sensor module(not shown). The sensor module (not shown) includes an accelerationsensor and/or a gyro sensor. The gyro sensor acquires information on thetilt of the electronic device 201, and the acceleration sensor acquiresinformation on a speed change of the electronic device 201. Theelectronic device 201 obtains information on a change in the location ofthe electronic device 201 when the electronic device 201 does notreceive information on the tilt or speed change of the externalelectronic device 202 from the first sensor 292 or the second sensor 294for a predetermined time, or receives information indicating null. Theprocessor 250 determines that the electronic device 201 has deviatedfrom the field of view of the camera 240 due to the movement of theelectronic device 201 without movement of the external electronic device202 when the information indicating the null is received from theexternal electronic device 202, or if the information is not receivedfor the predetermined time. Accordingly, the electronic device 201acquires information on a change in the location of the electronicdevice 201, and generates information on the change in the location ofthe external electronic device 202 based on the information on thechange in the location of the electronic device 201. For example, in thecase where the electronic device 201 moves in the field of view of thecamera 240 in the upper right direction, it is determined that theexternal electronic device 202 moves in the opposite direction (e.g.,the lower left direction). This is because a change in the location ofthe external electronic device 202 appears in a direction opposite tothe moving direction of the electronic device 201 based on the user'sfield of view (e.g., the display 230).

In operation 521, the processor 250 changes the display of the graphicobject. The processor 250 determines a location where the externalelectronic device 202 is predicted to move based on the receivedinformation about the change in the location of the external electronicdevice 202. The processor 250 changes the display of the graphic objectto correspond to the predicted location. According to an embodiment, theprocessor 250 changes a display of the graphic object so that a part ofthe graphic object is displayed and the remaining part is not displayed.For example, referring to the image 720 of FIG. 7, only a part of thegraphic object including the shape of a knife is displayed. According toanother embodiment, the processor 250 does not display the graphicobject. For example, referring to the image 740 of FIG. 7, the processor250 does not display the graphic object including the shape of a knife,but only an image related to multimedia content. The processor 250 stopsdisplaying the graphic object when the predicted location of theexternal electronic device 202 is outside a predefined distance based onthe information about the tilt information of the external electronicdevice 202 and/or the information about the speed change of the externalelectronic device 202. For example, when the graphic object includes ashape having a long length, the processor 250 sets a large size of apredefined distance. For another example, when the graphic objectincludes a shape having a short length, the processor 250 sets a smallsize of the predefined distance.

FIG. 6 illustrates an example of an external electronic device and agraphic object displayed to correspond to the external electronic deviceaccording to various embodiments.

Referring to FIG. 6A and FIG. 6B, the processor 250 acquires the cameraimage 610 using the camera 240. For example, the camera image 610includes an external electronic device 202, a user's hand for grippingthe external electronic device 202 and an image of the externalenvironment in which the user is located.

In various embodiments, the processor 250 displays the display image 620on the display 230. For example, the display image 620 includesmultimedia content and/or the graphic object 602 corresponding to theexternal electronic device 202. For example, the graphic object 602includes an image corresponding to the shape of a knife or a sword.

According to various embodiments, the processor 250 displays the graphicobject 602 and multimedia content in superimposed pattern. The processor250 renders or displays a background image related to multimediacontent. For example, when the content is related to tennis, thebackground image corresponds to an image related to a tennis court or atennis stadium. For example, the processor 250 displays the tennisracket on a location corresponding to the external electronic device 202in the background images associated with the tennis court or tennisstadium, so that the background image and/or the graphic object aresuperimposed. The electronic device 201 provides immersive servicesrelated to VR by displaying multimedia content and/or a graphic object602 superimposed on the multimedia content on the display 230.

FIG. 7 illustrates a change in display over time according to variousembodiments. Hereinafter, the images 710 to 760 displayed on the display230 include images corresponding to a user's field of view. In variousembodiments, the field of view of the camera 240 is the same as that ofthe display area of the images 710 to 760 displayed on the display.However, the field of view of the camera 240 is not limited thereto, andas illustrated in FIG. 4B, it is smaller or larger than the user's fieldof view.

Referring to FIG. 7A to FIG. 7F, according to various embodiments, theprocessor 250 displays the image 710. When the external electronicdevice 202 is located inside the field of view of the camera 240, theprocessor 250 obtains information about the location of the externalelectronic device 202 based on the image acquired by the camera 240. Theprocessor 250 displays the image 710 based on the information on thetilt of the external electronic device 202 and/or the locationinformation received from the external electronic device 202. Theprocessor 250 identifies the location of the electronic device 201located within the field of view of the camera 240, and displays agraphic object in a three-dimensional object (e.g., a knife or a sword).For example, referring to the image 710, the graphic object is displayedas an image including the shape of a knife. The image including theshape of the knife is displayed based on information on the location ofthe electronic device identified using the camera 240 and information onthe tilt received from the external electronic device 202, and as shownin the image 710, it is displayed to be inclined in the same manner asthe external electronic device 202 in the three-dimensional space.According to one embodiment, the processor 250 identifies the externalelectronic device 202 within the image obtained by the camera 240, anddisplays the knife on a location corresponding to the identifiedexternal electronic device 202. The processor 250 receives informationabout the tilting of the external electronic device 202 from theexternal electronic device 202, and based on the received informationabout the tilting, the tilting of the handle of the knife and the tiltof the knife blade are displayed. For example, the processor 250identifies an area brightly indicated by the LED in the image acquiredby the camera 240 and, based on information about the tilting of theexternal electronic device 202, it marks the handle part of the knifeand the blade part of the knife which are tilted at 45 degrees to theleft. The handle part of the knife is marked on the identified area.

According to various embodiments, the processor 250 displays the image720. When the external electronic device 202 deviates from the field ofview of the camera 240, the electronic device 201 stores locationinformation at the time point of deviation. For example, the processor250 detects the deviation when it fails to detect the brightness of athreshold illuminance value or higher in the image acquired by thecamera 240. For another example, the processor 250 identifies theexternal image of the external electronic device 202 or the image of themarker attached to the external electronic device 202 within theacquired image based on image processing of the image acquired by thecamera 240. The processor 250 determines that the external electronicdevice 202 is out of field of view of the camera 240 when it fails todetect the external image or the image for the marker. The processor 250requests the external electronic device 202 to transmit informationabout a change in the location of the external electronic device 202 inresponse to detecting the deviation. For example, referring to the image720, the user's hand is deviated from the camera's field of view.According to an embodiment, the processor 250 displays a part of thegraphic object. For example, if the external electronic device 202 doesnot move away by a predefined distance after the deviation time, theprocessor 250 displays only a part of the graphic object. Referring tothe image 720, the external electronic device 202 does not move adistance exceeding the length of the knife after deviating from thefield of view of the camera 240. Accordingly, the processor 250 does notdisplay the handle portion of the knife corresponding to the externalelectronic device 202 and displays a part of the graphic objectcorresponding to the blade portion of the knife.

According to various embodiments, the processor 250 displays the image730. The processor 250 receives information about a change in locationfrom the external electronic device 202. The processor 250 predicts thatthe external electronic device 202 has moved in the upper rightdirection by a certain distance based on the received information on thespeed change of the external electronic device 202 and/or information onthe tilt of the external electronic device 202. The processor 250changes the display of the inclined angle of the graphic object, asillustrated in the image 730 by reflecting the tilt information. Theprocessor 250 receives information of tilting of the external electronicdevice 202 while not displaying the handle portion of the knifecorresponding to the external electronic device 202, and displaying thegraphic object corresponding to the blade portion of the knife. Forexample, the processor 250 receives information on the tilt indicatingthat the external electronic device 202 is further tilted to the leftafter the external electronic device 202 is out of field of view of thecamera 240. Though the external electronic device 202 is not included inthe image acquired by the camera 240, the processor 250 changes theangle of the displayed blade portion based on the tilt information. Theprocessor 250 displays a graphic object corresponding to the blade ofthe knife to be tilted to the left based on the received tiltinformation.

According to various embodiments, the processor 250 displays the image740. The processor 250 receives information about a change in thelocation of the external electronic device 202 and does not display agraphic object corresponding to the external electronic device 202 onthe display 230. For example, the processor 250 acquires a predictedlocation to which the external electronic device 202 has been movedbased on data on the speed change of the external electronic device 202and/or data on tilt. When the predicted location is located farther thana predetermined distance, the processor 250 does not display a graphicobject including the shape of the knife. The predetermined distance is,for example, a distance equal to a length of a graphic object includingthe shape of the knife.

According to various embodiments, the processor 250 displays the image750 and/or the image 760. The processor 250 detects that the externalelectronic device 202 has moved into the field of view of the camera240. Accordingly, the processor 250 acquires information about thelocation of the external electronic device 202 through the camera 240and displays the graphic object. According to an embodiment, theprocessor 250 displays an image 750 corresponding to the externalelectronic device 202 at a time point when the external electronicdevice 202 enters into the field of view of the camera 240. Theprocessor 250 displays a graphic object at the time of re-entry based onthe information on the tilting of the external electronic device 202.For example, the processor 250 identifies the entry location of theexternal electronic device 202 based on the image acquired at the entrytime point. The processor 250 determines the degree of tilting of thegraphic object to be displayed on the entry location based on theinformation on the tilting of the external electronic device 202 amongthe information on the change in the received location. The processor250 displays the graphic object based on the determined tilting degreeand/or the entry location.

FIG. 8 illustrates an example for comparing the effect of correctionbased on information on a change in the location of an externalelectronic device according to various embodiments.

Hereinafter, the images 810 and 820 displayed on the display 230includes images corresponding to a user's field of view. In variousembodiments, the field of view of the camera 240 is the same as thefield of view of the display areas of the images 810 and 820 displayedon the display 230. However, the field of view of the camera 240 is notlimited thereto, and as illustrated in FIG. 4B, it is smaller or largerthan the user's field of view.

Referring to the image FIG. 8A, the image 810 corresponds to an image inwhich the display of the graphic object is not corrected by usinginformation on the change of the location. For example, the processor250 stores information corresponding to the deviation time point of theexternal electronic device 202. The processor 250 stores the location(hereinafter, the first location information) 815 and/or the slope(first sensor information) of the external electronic device withrespect to the deviation time point. Thereafter, when the externalelectronic device 202 moves into the field of view of the camera 240,the processor 250 detects the external electronic device 202. Theprocessor 250 displays a graphic object 813 for the entry time point ofthe external electronic device 202. For example, the processor 250 doesnot receive information about tilting from the external electronicdevice 202 after the deviation time point. The processor 250 displaysthe graphic object 813 according to the location (hereinafter, secondlocation information) 817 of the external electronic device 202 withrespect to the entry time point of the external electronic device 202.Since the processor 250 has not yet received the information on thetilting of the external electronic device 202 at the entry time point,the processor 250 displays the graphic object 813 based on the firstsensor information and/or the second location information. Thus, whileactual external electronic device 102 is inclined toward the upperright, the processor 250 displays the graphic object 813 according tothe slope corresponding to the deviation time point of the externalelectronic device 202.

Referring to the image FIG. 8B, the image 820 corresponds to an image inwhich the display of the graphic object is corrected by usinginformation on the change of the location. For example, the processor250 receives information on a change in the location of the externalelectronic device 202 from the external electronic device 202 during atime range from the deviation time point to the entry time point. Theprocessor 250 identifies information on the tilting of the externalelectronic device 202 corresponding to the entry time point, based onthe information on the change in the received location. Accordingly, theprocessor 250 displays the graphic object 823 based on the secondlocation information 827 corresponding to the entry time point of theexternal electronic device 202 and/or information on the tilting of theexternal electronic device 202 corresponding to the entry time point.For example, referring to FIG. 8A and FIG. 8B, graphic objects 811 and821 corresponding to the deviation time point of the external electronicdevice 202 is tilted toward the upper left at the deviation time point.Then, referring to FIG. 8B, the graphic object 823 displayed at theentry time point of the external electronic device 202 does not face theupper left, but it is displayed tilted to the top right to match thetilting of the external electronic device 202. In the above-describedembodiments, the processor 250 displays the graphic object naturally atthe entry time point by receiving information about a change in locationof the external electronic device 202 from the external electronicdevice 202 during a time from when the external electronic device 202deviates from the field of view of the camera 240 and then enters againthe field of view of the camera 240.

FIG. 9A illustrates an example of an operation of an electronic deviceaccording to various embodiments. Such an operation is performed by theelectronic device 201 illustrated in FIG. 2.

In operation 901, the processor 250 identifies the external electronicdevice 202 using the camera 240. According to an embodiment, theprocessor 250 acquires an image of the external environment through thecamera 240. The image of the external environment includes a number ofexternal objects. The processor 250 identifies the external electronicdevice 202 among the plurality of external objects. For example, theexternal electronic device 202 further includes a light emitting partfor generating light outside the housing. The processor 250 identifiesthe external electronic device 202 by identifying an area brighter thana predetermined illuminance in an image acquired through the camera.

In operation 903, when the external electronic device 202 is out of thecamera's field of view, the processor 250 receives one or more locationinformation from the external electronic device 202 which is acquiredduring a specified time range before the external electronic device 202enters a low power mode. According to an embodiment, the low power modecorresponds to a mode for reducing power consumption of the externalelectronic device 202. The external electronic device 202 determines toenter the low power mode and controls the amount of power provided to aplurality of components. For example, the external electronic device 202changes the operation cycle of the sensor module 290 into a long cycleto reduce power consumption, and decreases the number of operations ofthe sensor module 290. According to various embodiments, the low powermode is referred to by various terms including a sleep mode, a sleepingmode, an inactive mode, an inactive state, a deactivation state, and thelike. According to an embodiment, the external electronic device 202performs a low power mode in which the external electronic device 202does not perform a data acquisition operation by the sensor module 290during the specified time range. When the external electronic device 202acquires at least one of the information on the tilt obtained by thefirst sensor 292 and/or information on the speed change obtained by thesecond sensor 294, the external electronic device 202 does not enter thelow power mode. When the external electronic device 202 does not enterthe low power mode, the external electronic device 202 monitors theacquisition of the above at least one information during the abovespecified time range from a time point corresponding to the acquisitionof the at least one information. According to various embodiments, theexternal electronic device 202 acquires one or more location informationduring the specified time range. The sensor module 290 is activated fromthe specified time range, that is, from the last time a user input isreceived, to a time when the low power mode is performed. The sensormodule 290 receives information on the tilting of the externalelectronic device 202 corresponding to a time range from the time whenthe user input is received to the time when the low power mode isperformed.

In operation 905, the processor 250 receives a display request todisplay a graphic object associated with the external electronic device202. The display request is based on a change in the operation mode ofthe external electronic device 202. For example, the processor 250detects that the operation mode is changed from an inactive mode to anactive mode, and in response to the detection, receives a request signalrequesting display of a graphic object. The request signal correspondsto a signal received from another electronic device (e.g., the secondexternal electronic device 202). According to another embodiment, therequest signal includes a control signal received from elements of theexternal electronic device 202. For another example, the display requestof the graphic object is generated by detecting that the data value ofthe external electronic device 202 acquired by the first sensor 292exceeds the threshold value. For example, when the external electronicdevice 202 detects a motion exceeding a predetermined threshold, thedisplay request for the graphic object is transmitted to the electronicdevice 201. For another example, the external electronic device 202 doesnot generate the display request when the detected change amount ofmotion does not exceed a predetermined threshold (e.g., vibration due tocarelessness of the user is a case). For another example, the externalelectronic device 202 further includes a grip sensor (not shown). Theexternal electronic device 202 generates the display request when thegrip sensor detects a grip of the user.

In operation 907, the processor 250 determines a location to display thegraphic object based on the one or more location information. The one ormore location information includes information on a location where theexternal electronic device 202 deviates from the field of view of thecamera 240, the first sensor information corresponding to a time pointwhen entering the low power mode, and the second sensor informationcorresponding to a time point when releasing from the low power mode.

FIG. 9B illustrates an example of an operation of an electronic deviceaccording to various embodiments. Such an operation is performed by theelectronic device 201 illustrated in FIG. 2.

In operation 911, the processor 250 obtains information about thelocation of the external electronic device 202 interlocked with theelectronic device 201. In operation 913, the processor 250 receivesinformation about the tilting of the external electronic device 202. Inoperation 915, the processor 250 provides a graphic object (visualobject) for representing the external electronic device 202 togetherwith multimedia content. The operations 911 to 915 correspond tooperations 511 to 515 shown in FIG. 5B, respectively.

In operation 917, the processor 250 determines that the sleep mode hasbeen entered. The sleep mode corresponds to a mode for reducing powerconsumption of the external electronic device 202. According to anembodiment, when the user input is not received for a predeterminedlength of time, the processor 250 determines that entering the sleepmode has been performed.

In operation 919, the processor 250 receives location information and/ortilt information of the external electronic device 202 corresponding toa time point of entering the sleep mode. For example, when the processor250 does not receive a user input for a predetermined length of time,the processor 250 determines that a low power mode has been entered, andcontrols to receive information related to the external electronicdevice 202 corresponding to the entry time point for the low power mode.

In operation 921, the processor 250 identifies that the sleep mode isreleased. The sleep mode is released by receiving a wake-up signal fromthe external electronic device 202. The wake-up signal is referred to invarious terms such as a paging signal, an activation signal and thelike. For example, the wake-up signal is generated when a motion of theexternal electronic device 202 exceeding a predefined threshold value ofthe external electronic device 202 is detected. For another example, theexternal electronic device 202 further includes a grip sensor (notshown). The external electronic device 202 generates the wake-up signalbased on obtaining data, from the grip sensor, indicating that the userhas gripped the external electronic device 202.

In operation 923, the processor 250 predicts the location of theexternal electronic device 202 based on the information about thetilting of the external electronic device 202 corresponding to therelease time point of the sleep mode, the information about the tiltingof the external electronic device 202 corresponding to the time point ofentering the sleep mode and/or the information on the location of theexternal electronic device 202 corresponding to a time point whenentering the sleep mode. According to an embodiment, the processor 250predicts a change in the location of the external electronic device 202by comparing a difference between information corresponding to the entrytime of the sleep mode and/or information corresponding to the releasetime among information on the tilting of the external electronic device202. The processor 250 obtains a location predicted to be located at thetime of the release of the sleep mode by applying the information on thechange of the predicted location to the location information of theentry time of the sleep mode.

According to various embodiments, the processor 250 predicts thelocation based on the information on the tilting of the electronicdevice 201. For example, the electronic device 201 moves between thetime point of entering the sleep mode and the time point of releasingthe sleep mode. In order to reflect the movement of the electronicdevice 201, the processor 250 compares the information on the tilting ofthe electronic device 201 at the time when the external electronicdevice 202 enters the sleep mode and/or the information on the tiltingof the electronic device 201 at the time when the external electronicdevice 202 releases the sleep mode, and thereby the processor 250obtains information about the movement (motion) of the electronic device201.

In operation 925, the processor 250 determines whether the predictedlocation of the external electronic device is included in the field ofview of the camera. The processor 250 displays a graphic object at thelocation of the external electronic device 202 predicted in operation929 when the predicted location of the external electronic device 250 iswithin an area for the field of view of the camera 240. The processor250 displays a graphic object for guiding the predicted location inoperation 927 when the predicted location of the external electronicdevice is outside the area of the field of view of the camera 240. Thegraphic object is not a graphic object corresponding to the externalelectronic device 202. For example, the graphic object includes anarrow-shaped image. For another example, the graphic object includes animage identical or similar to the shape of the external electronicdevice 202.

FIG. 10 illustrates an example of a graphic object that guides thelocation of an external electronic device according to variousembodiments. Hereinafter, the images 1010 and 1020 displayed on thedisplay 230 include images corresponding to a user's field of view. Invarious embodiments, the field of view of the camera 240 is the same asthe field of view of the images 1010 and 1020 displayed on the display230. However, the field of view of the camera 240 is not limitedthereto, and as illustrated in FIG. 4B, it is smaller or larger than theuser's field of view.

Referring to FIG. 10, the processor 250 displays an image 1010 relatedto multimedia content on the display 230. The image 1010 related withthe multimedia content includes at least one of omnidirectional imagesor 3D images. In various embodiments, the processor 250 displays agraphic object for guiding the location of the external electronicdevice 202. For example, the processor 250 displays a graphic object1012 in the shape of an arrow. The arrow-shaped graphic object 1012 isdisplayed to be superimposed on an image related to the displayedmultimedia content. According to an embodiment, the processor 250changes the display of the arrow-shaped graphic object 1012. Forexample, when the external electronic device 202 moves away from theelectronic device 201, the processor 250 reduces the thickness of thegraphic object 1012 in the shape of an arrow. For another example, whenthe distance between the external electronic device 202 and theelectronic device 201 decreases, the processor 250 increases thethickness of the graphic object 1012 in the form of an arrow. In anotherembodiment, the processor 250 changes the color of a graphic object. Theprocessor 250 changes the color of the graphic object based on the colorof multimedia content displayed on the display 230. For example, theprocessor 250 determines a display area of the graphic object 1012 inthe shape of an arrow, and identifies some areas of an image related tomultimedia content corresponding to the determined area. The processor250, in order to improve the discrimination power for the graphic object1012, identifies the color of the identified area, and displays thegraphic object 1012 in the shape of an arrow with the complementarycolor of the identified color.

In various embodiments, the processor 250 displays a visual effect 1022.The processor 250 displays the image 1020 associated with the multimediacontent on which the visual effect 1022 is superimposed. For example,the processor 250 displays the visual effect 1022 in a boundary surfaceof a direction to which the external electronic device 202 is predictedto be located among the four boundary surfaces making the boundary ofthe display 230. The visual effect 1022 includes an effect ofhighlighting the boundary surface. Although not shown, the processor 250includes a visual effect related to blur for blurring the boundarysurface. In the above-described embodiment, the graphic object 1012 hasbeen described as including an arrow-shaped image, but is not limitedthereto. For example, the graphic object 1012 includes an image of thesame shape with or similar shape to the external electronic device 202.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smartphone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

It should be appreciated that various embodiments of the presentdisclosure and the terms used therein are not intended to limit thetechnological features set forth herein to particular embodiments andinclude various changes, equivalents, or replacements for acorresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar orrelated elements. It is to be understood that a singular form of a nouncorresponding to an item may include one or more of the things, unlessthe relevant context clearly indicates otherwise. As used herein, eachof such phrases as “A or B,” “at least one of A and B,” “at least one ofA or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least oneof A, B, or C,” may include any one of, or all possible combinations ofthe items enumerated together in a corresponding one of the phrases. Asused herein, such terms as “1st” and “2nd,” or “first” and “second” maybe used to simply distinguish a corresponding component from another,and does not limit the components in other aspect (e.g., importance ororder). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

As used herein, the term “module” may include a unit implemented inhardware, software, or firmware, and may interchangeably be used withother terms, for example, “logic,” “logic block,” “part,” or“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 140) including one or more instructions that arestored in a storage medium (e.g., internal memory 136 or external memory138) that is readable by a machine (e.g., the electronic device 101).For example, a processor (e.g., the processor 120) of the machine (e.g.,the electronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of theprocessor. This allows the machine to be operated to perform at leastone function according to the at least one instruction invoked. The oneor more instructions may include a code generated by a complier or acode executable by an interpreter. The machine-readable storage mediummay be provided in the form of a non-transitory storage medium. Wherein,the term “non-transitory” simply means that the storage medium is atangible device, and does not include a signal (e.g., an electromagneticwave), but this term does not differentiate between where data issemi-permanently stored in the storage medium and where the data istemporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities. According to various embodiments, one or more ofthe above-described components may be omitted, or one or more othercomponents may be added. Alternatively or additionally, a plurality ofcomponents (e.g., modules or programs) may be integrated into a singlecomponent. In such a case, according to various embodiments, theintegrated component may still perform one or more functions of each ofthe plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components beforethe integration. According to various embodiments, operations performedby the module, the program, or another component may be carried outsequentially, in parallel, repeatedly, or heuristically, or one or moreof the operations may be executed in a different order or omitted, orone or more other operations may be added.

1. An electronic device, comprising: one or more cameras having a designated field of view; a display; a communication circuitry; and a processor, wherein the processor is configured to, identify an external electronic device among one or more external objects included in the designated field of view using the camera, based on first location information of the external electronic device identified based at least on image information obtained using the camera, display a graphic object corresponding to the external electronic device on the display, and when the external electronic device is out of the designated field of view, display the graphic object on the display based on second location information of the external electronic device identified using the camera before the external electronic device is out of the designated field of view and information related to the movement of the external electronic device received from the external electronic via the communication circuitry after the external electronic device is out of the designated field of view.
 2. The electronic device according to claim 1, wherein the external electronic device further comprises a first sensor comprising a gyro sensor; and a second sensor including an acceleration sensor, and wherein the information related to the movement of the external electronic device comprise at least one of data related to tilting of the external electronic device obtained by the first sensor or data related to a speed change of the external electronic device obtained by the second sensor.
 3. The electronic device according to claim 1, wherein the external electronic device further comprises a light emitting part including one or more light emitting diode (LED), and wherein the processor is configured to acquire an image including at least a portion of the light emitting part using the camera, estimate a location related to the light emitting part based at least on the image calculation for the image, and determine the first location information related to the external electronic device based at least on the estimated location.
 4. The electronic device according to claim 2, wherein the processor is configured to, when the external electronic device is located within the designated field of view, identify a display location of the graphic object based at least on the first location information and data related to the tilt of the external electronic device, when the external electronic device is out of the designated field of view, identify the display location of the graphic object based at least on the second location information and a third location information, and the third location information includes vector information obtained based on data related to the tilt of the external electronic device and data related with a change in speed of the external electronic device, and display the graphic object based on the identified display location.
 5. The electronic device according to claim 1, wherein the processor is configured to display the graphic object based on multimedia content related to virtual reality, the first location information, and the second location information provided by the electronic device.
 6. The electronic device according to claim 1, wherein the processor is configured to, if the external electronic device is not identified within the designated field of view, receive, from the external electronic device, one or more third location information acquired during the designated time range before the external electronic device enters the low power mode, and determine a display location of the graphic object to be displayed on the display based at least on the one or more third location information when the display request of the graphic object is confirmed.
 7. The electronic device according to claim 1, wherein the processor is configured to, identify that the external electronic device has entered the sleep mode, receive the third location information of the external electronic device corresponding to the entry time point and information related to the tilting of the external electronic device, identify that the sleep mode is released, receive information related to the tilting of the external electronic device corresponding to the release time point, display a second graphic object for indicating the location of the external electronic device based at least on the third location information of the external electronic device and information related to the tilting of the external electronic device corresponding to the entry time point, and information related to the tilting of the external electronic device corresponding to the release time point.
 8. The electronic device according to claim 7, wherein the second graphic object includes an image of the same or similar shape with the external electronic device or an image of an arrow shape.
 9. A method of an electronic device, comprising: identifying an external electronic device among one or more external objects included in the designated field of view using one or more cameras having a designated field of view; displaying a graphic object corresponding to the external electronic device on a display based on first location information of the external electronic device identified based at least on image information obtained through the camera; and when the external electronic device is out of the designated field of view, displaying the graphic object on the display based on second location information of the external electronic device identified using the camera before the external electronic device is out of the designated field of view and information related to the movement of the external electronic device received from the external electronic device via a communication circuitry after the external electronic device is out of the designated field of view.
 10. The method according to claim 9, wherein the information related to the movement of the external electronic device includes at least one of data related to tilting of the external electronic device obtained by the first sensor included in the external electronic device, or data related with a change in speed of the external electronic device obtained by the second sensor included in the external electronic device.
 11. The method according to claim 10, further comprising, when the external electronic device is located within the designated field of view, identifying a display location of the graphic object based at least on the first location information and the data related to the tilting of the external electronic device; when the external electronic device is out of the designated field of view, identifying the display location of the graphic object based at least on the second location information and the third location information; and based on the identified display location, displaying the graphic object, wherein the third location information includes vector information obtained based on the data related to the tilting of the external electronic device and the data related to the speed change of the external electronic device.
 12. The method according to claim 9, further comprising, obtaining an image including at least a portion of a light emitting part included in the external electronic device using the camera; estimating a location related to the light emitting part based at least on an image operation on the image; and determining the first location information related to the external electronic device based at least on the estimated location.
 13. The method according to claim 9, the displaying the graphic object on the display includes displaying the graphic object based on multimedia content related to virtual reality, the first location information, and the second location information.
 14. The method according to claim 9, further comprising, when the external electronic device is not identified within the designated field of view, receiving one or more third location information, from the external electronic device, acquired during the designated time range before the external electronic device enters a low power mode; and when the display request of the graphic object is confirmed, determining a display location of the graphic object to be displayed on the display based at least on the one or more third location information.
 15. The method according to claim 9, further comprising, identifying that the external electronic device has entered a sleep mode; receiving third location information of the external electronic device corresponding to the entry time point and information related to tilting of the external electronic device; identifying that the sleep mode has been released; receiving information related to tilting of the external electronic device corresponding to the release time point; and displaying a second graphic object for indicating the location of the external electronic device at least based on the third location information of the external electronic device and information related to the tilting of the external electronic device corresponding to the entry time point, and information related to the tilting of the external electronic device corresponding to the release time point.
 16. An electronic device, comprising: a housing; a memory; a camera having a designated field of view and visually exposed through at least a portion of a first surface of the housing; a display exposed through at least a portion of a second surface, of the housing, different from the first surface; a communication module; and a processor, wherein the processor is configured to: obtain information related to a location of an external electronic device interlocked with the electronic device using the camera, receive information related to tilting of the external electronic device from the external electronic device using the communication module, identify a graphic object based on at least one of the information related to the location or the information related to tilting, provide the graphic object for representing the external electronic device with multimedia content related to virtual reality using the display, identify that the external electronic device deviates from the designated field of view of the camera using the camera, in response to identifying the deviation, transmit a signal for requesting the external electronic device to transmit information related to a change in the location of the external electronic device, and change a display of the graphic object based on at least one of the information related to the location of the external electronic device at a deviation time obtained using the camera, the information related to tilting of the external electronic device, or the information related to the change in the location of the external electronic device.
 17. The electronic device according to claim 16, wherein the information related to the change in the location of the external electronic device is determined based on data obtained by a sensor included in the external electronic device, and wherein the sensor includes at least one of an acceleration sensor or a gyro sensor.
 18. The electronic device according to claim 16, wherein the processor is configured to activate an acceleration sensor in response to identifying that the external electronic device deviates from the designated field of view of the camera.
 19. The electronic device according to claim 16, wherein the processor is configured to obtain a location to which the external electronic device is predicted to have moved based at least on the information related to the location of the external electronic device at the deviation time, the information related to tilting of the external electronic device, and the information related to the change in the location of the external electronic device.
 20. The electronic device according to claim 16, wherein the graphic object is displayed superimposed on the multimedia content related to the virtual reality, and wherein the multimedia content includes content for providing virtual reality service. 